Demonstration and Discussion of Concept

This code serves as a demonstration and discussion of the proposed concept.

Key Decisions:

• Avoiding runnable: I skip using runnable for a config and configurable for a processor. Also, all steps will operate directly on HF:datasets.Dataset.
• Introducing Apply Interface: A new interface, Apply, will be implemented. It will be HF-centric and allow applying any required processing on each dataset shard be it filter map or any other step supported by datasets.
• Step Specification Approach: The approach used in datasets is followed, where step types are specified in the config file by type field.
• Aggregator Class: An aggregator class will sequentially invoke processing steps, in preparator config class it is called processors.
• Hollow Doc Length Filter: Implemented an empty doc lenght filter step to demonstrate how steps could look like.

Usage:

To prepare a dataset, simply call:

dataset = self._config.processors.apply(dataset)

config would be something like this:

processors:
  steps:
    -
     type: length_filter
     field: text
     min_length_chars: 100
     max_length_chars: 100000
    - ...

@jlamypoirier, @tscholak What do you think?

Hi @bigximik, thanks for putting this together. I appreciate the careful thinking you've put in here!

However, let's simplify significantly. The goal isn't to design a general, modular pipeline system. It's just about adding these very specific cleaning filters. We already know exactly what filters we want and in what order.

Here's what I'd suggest:

• Just write a single function clean_dataset(dataset, config) that explicitly calls each filter step in sequence.
• Each filter can be toggled with a simple boolean in the config. No need for abstract registries or complicated type-based dispatch.
• Each filter is just a simple, testable function that uses .filter or .map directly on the HuggingFace dataset.
• Let's keep the configuration as simple as possible (just booleans and thresholds).

We can always refactor if more complexity is actually required down the line, but let's get this feature shipped quickly and cleanly first.

Can you please move forward by just implementing the concrete filters directly?

Created basic implementation based on feedback.

• Word splitting and number detection implemented using regex.
• Binary data detection is based on non-printable characters.
• Presidio Model Management: Needs proper handling of model downloading and loading. Currently, it relies on SpaCy’s internal package installation, which won’t work in a frozen Docker image.
• ClamAV Integration: Needs a proper implementation. The chat-gpt suggested approach does not work. Proper implementation requires database download, loading, and file scanning, either via an outdated Python wrapper or a direct approach.
• Unit tests now cover all processors except ClamAV.
• Need to implement test skipping for PII and malware detection processors.
• Integration testing with prepare is pending.

Next Steps

• Implement model download and loading management for Presidio.
• Implement multiprocessing for Presidio engine.
• Implement ClamAV database management and scanning.
• Add test skipping for specific processors.
• Verify integration with prepare.